A computer system is tested several times over the course of its life cycle, starting with the computer program initial design and being repeated every time the computer system is modified. Testing of computer systems (e.g., software/computer program) is usually conducted by a test engineer who identifies defects by manually running a product through a defined series of steps and observing a result after each step. The testing has become a time consuming and extremely expensive process because the series of steps are intended to both thoroughly exercise product functions as well as re-execute scenarios that have identified problems.
A known method for computer program testing is regression testing by which is meant going back to earlier performed tests and doing them again to control a developing computer program's behavior such that it is unchanged from a last testing occasion. A current test result must be identical to a last recorded result. Regression testing can also be seen as any type of computer program testing that seeks to uncover regressions, or any type of software testing that seeks to find side effects. Side effects may be that when a bug itself is fixed the fix then might create other bugs. Fixing a minor bug may result in a major side effect and a bug fix for a high severity defect may have no or just a minor side effect. By regression it is usually meant a return to a former, usually (or generally) worse condition. As an example, such regressions occur whenever a previously working computer program functionality stops working as it should. Typically, regressions occur due to unintended program changes.
Existing well known methods of regression testing include rerunning previously run tests and evaluating whether previously fixed faults have re-emerged, i.e., repeat to happen. Usually hundreds of tests are performed to ensure that old features of a computer program are intact. Success in regression testing relies on regressions being found, isolated, and corrected quickly, in order to prevent code instabilities from aggregating and leading to quality deficiencies.
Experience has shown that as a computer program is corrected, emergence of new and/or reemergence of old faults are/is quite common. Correcting a fault in one area inadvertently might cause a computer program fault in another area. In addition, it has often been the case that when some feature is redesigned, the same mistakes that were made in the original implementation of the feature were made in the redesign as well.
Regression testing may be performed manually using programming techniques, e.g., by manually running specific tests recorded, or test use cases, to expose a certain fault. This may be done regularly by, for example, an administrator after subsequent changes to the computer program. Running such tests manually consumes a huge amount of time and resources. Thus, automation of regression testing may save money and time and increase the quality of the tested program, since testing may be determined to be performed immediately after a change is done to the computer program.
There are several existing technologies (automated systems, e.g., capture/replay tools, or robots) for automatic regression testing of computer programs, which allow a testing environment to execute all the regression testing automatically, which is to say without interaction with a human user. In some working areas, such as stock trading, automated systems are set up to automatically rerun some, or all, regression tests at specified time intervals and report any failures. These failures may further imply that there are one or more regressions in the computer program. Usually, automatic regression testing is performed once each night, or once every day.
Conventionally, automatic regression testing may be performed for tests at lower levels of integration (e.g., unit tests), but there is a need to automate all tests including all levels of integration (i.e., unit, integration, system, and acceptance testing). In general, a program is divided in several units where each unit might belong to a certain group of units. Each group is then associated with certain tests that should be used for regression testing of that group.
Grouping of units is typically performed based on unit characteristics. As an example, stock trading systems require high computer program quality and therefore it is considered important to be able to perform regression testing in such systems at all levels (e.g., trading clients, market management clients, system operation clients, etc.).
A drawback of the above-described automated systems is that they are sensitive to the technology used by clients. Some automated systems are dependent on a working operating system; other solutions are dependent on a system language; some of the known solutions are dependent on a web browser; some known solutions are not suitable, e.g., for certain programming languages; and, some known solutions simply cannot be easily maintained for regression testing.
Further to the above, regression tests are required to be very simple and reliable and therefore they are usually manually coded. An additional disadvantage with the existing automated systems for automatically regression testing is that a change in a computer program, a working progress, an interface of a computer program, or a test generally requires updating several tests stored in a test database. This means that huge amount of scripts need to be manually coded and updated upon a change in the computer program to be used when regression testing or when the computer program itself is tested. The change of such a computer program may for example be a change in a graphical user interface (GUI).